Systems and methods for generating and presenting virtual experiences

ABSTRACT

Various embodiments facilitate the creation and presentation of a virtual experience. In one embodiment, the virtual experience is assembled from user model data corresponding to a three-dimensional representation of a user, user movement data corresponding to at least one movement characteristic of the user, user voice data corresponding to at least one vocal characteristic of the user, environment data corresponding to a three-dimensional representation of a location, and event data corresponding to a captured event at the location. The virtual experience is a virtual recreation of the captured event at the location, with the three-dimensional representation of the user, the vocal characteristic of the user, and the movement characteristic of the user inserted into the captured event.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Patent application Ser. No.15/093,912, filed Apr. 8, 2016, entitled “SYSTEMS AND METHODS FORGENERATING AND PRESENTING VIRTUAL EXPERIENCE,” the entire disclosure ofwhich is herein incorporated by reference.

TECHNICAL FIELD

This disclosure relates to virtual experiences and methods of creatingand presenting the same.

BACKGROUND

A recording of an event typically captures visual information with acamera and audio information with a microphone. The visual informationmay be captured in two dimensions or in three dimensions. Similarly, theaudio information may be captured in mono, stereo, or surround sound.

The recorded visual information and audio information may be used topresent a recreation of the event. The recreation of the event istypically from the point-of-view of the camera, and the people, places,things, sounds, and movements that occurred during the event are fixedbased on what was captured by the camera and microphone.

SUMMARY

The present disclosure relates to systems, methods, and apparatus forfacilitating the creation and presentation of a virtual experience.According to one embodiment, a virtual experience system, includes auser model storage configured to store user model data corresponding toa three-dimensional representation of a user; a user movement storageconfigured to store user movement data corresponding to at least onemovement characteristic of the user; a user voice storage configured tostore user voice data corresponding to at least one vocal characteristicof the user; an environment storage configured to store environment datacorresponding to a three-dimensional representation of a location; anevent storage configured to store event data corresponding to a capturedevent at the location; a receiver coupled to the user model storage, theuser movement storage, the user voice storage, the environment storage,and the event storage, the receiver configured to assemble a virtualexperience based at least in part on the user model data, the usermovement data, the user voice data, the environment data, and the eventdata; and a display device coupled to the receiver, the display deviceconfigured to display the virtual experience to a user, the user beingable virtually recreate the captured event at the location, with thethree-dimensional representation of the user, the vocal characteristicof the user, and the movement characteristic of the user inserted intothe captured event.

In some embodiments, the receiver is configured to assemble the virtualexperience such that the three-dimensional representation of the user,the vocal characteristic of the user, and the movement characteristic ofthe user replace a person who was physically present at the capturedevent. In some embodiments, the system further includes a user modelingdevice configured to capture the user model data, the user voice data,the user movement data, or a combination thereof. In some embodiments,the system further includes an event capture device configured tocapture the event data, the environment data, or a combination thereof.

In some embodiments, the virtual experience includes virtual merchandiseinserted into the virtual experience. In some embodiments, the userpurchases the virtual merchandise. In some embodiments, the virtualmerchandise replaces physical merchandise present at the captured event.In some embodiments, the captured event is associated with a real-lifeevent captured at a physical location. In some embodiments, the virtualexperience with the three-dimensional representation of the user, thevocal characteristic of the user, and the movement characteristic of theuser inserted into the captured event is stored as a new event. In someembodiments, the receiver is further configured to assemble the virtualexperience based at least in part on second user model data, second usermovement data, and second user voice data, such that multiple users areinserted into the captured event.

It is to be understood that both the foregoing summary and the followingdetailed description are for purposes of example and explanation and donot necessarily limit the present disclosure. The accompanying drawings,which are incorporated in and constitute a part of the specification,illustrate subject matter of the disclosure. Together, the descriptionsand the drawings serve to explain the principles of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The same number represents the same element or same type of element inall drawings.

FIG. 1 is an overview block diagram illustrating an example of data flowfor a virtual experience process according to principles disclosedherein.

FIG. 2 is a block diagram illustrating an example of a virtualexperience system for practicing example embodiments of the virtualexperience process according to principles disclosed herein.

FIG. 3 is a diagram illustrating an example of a user modeling devicefor practicing example embodiments of the virtual experience processaccording to principles disclosed herein.

FIG. 4 is a diagram illustrating an example of an event capture devicefor practicing example embodiments of the virtual experience processaccording to principles disclosed herein.

FIG. 5 is a block diagram illustrating an example of virtual merchandisefor the virtual experience according to principles disclosed herein.

FIG. 6 is a block diagram illustrating an example of virtualenvironments for the virtual experience according to principlesdisclosed herein.

FIG. 7 is a schematic illustrating an example of the user connected tothe virtual experience system according to principles disclosed herein.

DETAILED DESCRIPTION OF THE DRAWINGS

The technology described herein is related to U.S. patent applicationSer. No. 14/449,013, filed on Jul. 31, 2014, which is incorporated byreference herein in its entirety.

The various embodiments described herein generally provide apparatus,systems and methods which facilitate the creation and presentation of avirtual experience. The virtual experience is a virtual recreation of acaptured event with a three-dimensional representation of a userinserted into the captured event.

FIG. 1 is an overview block diagram illustrating an example of data flowfor a virtual experience process 20 according to principles disclosedherein. In this example, the virtual experience process 20 includes auser 22, a virtual experience 24, user model data 26, user movement data28, user voice data 30, environment data 32, and event data 34.

The user 22 is presented the virtual experience 24 that the user 22 mayparticipate and interact within. As will be further discussed herein,the user 22 may view the virtual experience 24 by using a headset 52.

The virtual experience 24 is created using one or more of the variousdata sources 26-34. In some embodiments, the virtual experience process20 uses the user model data 26, user movement data 28, user voice data30, environment data 32, and event data 34 to generate the virtualexperience 24.

The user model data 26 includes a three-dimensional model of the user22. The three-dimensional model may include the user's entire body, orportions of the user's body, such as arms, hands, legs, and/or feet. Theuser model data 26 may include accurate measurements of the user's bodydimensions (e.g., body size, head size, foot size, etc.), as well asfeatures of the user's appearance (e.g., haircut, clothes, earrings,glasses, etc.). The three-dimensional model may be generated by scanningthe user's body with various three-dimensional scanning techniques knownin the art. The user model data 26 allows a virtual three-dimensionalversion of the user 22 to be recreated in the virtual experience 24.

The user movement data 28 includes styles or patterns of movement of theuser 22. The styles or patterns of movement may include a way that theuser 22 walks, runs, jumps, dances, laughs, and/or other commonmovements. The user movement data 28 may be generated by recording theuser 22 performing sample movements, and then extrapolating thosemovements to the a movement model. For example, the user's movements maybe extrapolated to a three-dimensional wire-frame model of a human. Theuser movement data 28 allows a virtual version of the user 22 recreatedin the virtual experience 24 to move in a similar way as the user 22.

The user voice data 30 includes a model of the sound of the user'svoice. The model of the sound of the user's voice may be generated byrecording the user 22 saying sample words and/or phrases. Thecharacteristics of the user's voice may then be modeled using one ormore vocal modeling techniques known in the art. The user voice data 30allows a virtual version of the user 22 recreated in the virtualexperience 24 to have a similar voice as the user 22.

The environment data 32 includes a three-dimensional model of areal-world or virtual location. For example, the three-dimensional modelmay be of a room, stadium, park, museum, street, and/or otherenvironments. The three-dimensional model of the location may begenerated by scanning the real-world location with variousthree-dimensional scanning techniques known in the art. Alternatively orin addition, the real-world or virtual location may be generated by adesigner. The environment data 32 allows a virtual three-dimensionalversion of a location to be recreated in the virtual experience 24

The event data 34 includes a recording of an event. The captured eventmay occur in the location associated with the environment data 32, or inanother location. The event may include the user 22, or another user. Insome examples, the other user may be an actor, athlete, politician, orother public figure. The event data 34 may include the movements andpoints-of-view of the user participating in the event. The event data 34may also include sounds heard by the user participating in the event,and words spoken by the user participating in the event. The event data24 allows the event to be recreated for the user 22 as the virtualexperience 24.

The virtual experience 24 is a recreation of the captured eventcontained in the event data 34. The virtual experience 24 may take placein the location of the captured event, or in another location providedby the environment data 32. The virtual experience 24 replaces the userwho originally participated in the captured event with a virtualrecreation of the user 22. The virtual recreation of the user 22 isgenerated based on the user model data 26, the user movement data 28,and the user voice data 30. This allows the user 22 to experience thecaptured event as if they were the user that originally participated inthe event.

After doing so, the new event with the user 22 may be captured,recorded, stored, and shared with other users as if the user 22participated in the original event. For example, the user 22 couldexperience skydiving by taking the place of a real skydiver who recordedthe event. The user 22 could then save and share this virtual experience24 with their friends. As another example, an actor could record anevent on the set of a movie. The user 22 could can then take the placeof the actor in that event, and virtually experience what it feels liketo be an actor.

In some embodiments, the virtual experience 24 may provide medicalanalysis of a user 22. For example, the virtual experience 24 mayevaluate the user's body structure, motion, and health by analyzing theuser model data 26, the user movement data 28, and/or the user voicedata 30.

In some embodiments, multiple users may participate in a single virtualexperience 24 in real-time. For example, the event may be a sportingevent recorded by players. Then two users may place themselves at twodifferent positions on the sports team. As another example, the eventmay be an educational lesson. Multiple users may then participate in theeducational lesson and interact with each other. For example, one user22 may be inserted into a biology lab environment with two other usersthat are also inserted into the biology lab environment. However, eachof the users may be in physically different locations. In this labenvironment, the users can dissect a frog together, for example, and thevirtual actions of the users can be evaluated by the virtual experience24 to determine the results. For example, if a user's virtual actionsinclude slicing off the frog's leg, the frog's leg will appear to falloff in the virtual experience 24. In some examples, the virtualexperience 24 may also notify the user if they are doing something wrongin the virtual experience 24.

In some embodiments, another user without specific user model data 26,user movement data 28, or user voice data 30 may also utilize thevirtual experience 24. In these embodiments, the user participates inthe virtual experience 24 using a generic virtual user model, or with avirtual user model of the person who originally recorded the event.

In some embodiments, the virtual experience 24 may be monetized. Forexample, users who record events could receive fees from other users whoplace themselves in the events through the virtual experience 24. Inaddition, production companies may create virtual experiences 24, andthen charger users to place themselves in the experiences.

FIG. 2 is a block diagram illustrating an example of a virtualexperience system 36 for practicing example embodiments of the virtualexperience process 20 according to principles disclosed herein. In thisexample, the virtual experience system 36 includes a user model storage38, user movement storage 40, user voice storage 42, environment storage44, event storage 46, a receiver device 48, and a headset 52. The usermodel data 26, user movement data 28, user voice data 30, environmentdata 32, and event data 34 described in reference to FIG. 1 are recordedin the user model storage 38, user movement storage 40, user voicestorage 42, environment storage 44, and event storage 46, respectively.

The user model storage 38, user movement storage 40, user voice storage42, environment storage 44, and event storage 46 are coupled to thereceiver 48. In some embodiments, the user model storage 38, usermovement storage 40, user voice storage 42, environment storage 44, andevent storage 46 may be coupled to the receiver 48 through a server ornetwork connection. For example, the user model storage 38, usermovement storage 40, user voice storage 42, environment storage 44, andevent storage 46 may be part of a cloud computing architecture which thereceiver 48 accesses to retrieve data for the virtual experience 24.

The receiver 48 is configured to access the user model storage 38, usermovement storage 40, user voice storage 42, environment storage 44, andevent storage 46 and to assemble the virtual experience 24 from the usermodel data 26, user movement data 28, user voice data 30, environmentdata 32, and/or event data 34. The receiver 48 may access the user modelstorage 38, user movement storage 40, user voice storage 42, environmentstorage 44, and/or event storage 46 periodically, upon notification ofdata being updated, or on-demand.

The receiver 48 may be configured to obtain one or more of the usermodel data 26, user movement data 28, user voice data 30, environmentdata 32, and/or event data 34 from a server. Alternatively or inaddition, in some embodiments, the receiver 48 may be configured toobtain one or more of the user model data 26, user movement data 28,user voice data 30, environment data 32, and/or event data 34 through asatellite or coaxial cable connection. Upon obtaining the data, thereceiver 48 is configured to combine the user model data 26, usermovement data 28, user voice data 30, environment data 32, and eventdata 34 to create the virtual experience 24. The receiver 48 may obtainthe data periodically, upon notification of data being updated, oron-demand. In another embodiment, a remote server assembles virtualexperience 24, and the receiver 48 obtains the assembled virtualexperience 24 from the remote server.

The receiver 48 may be any suitable converter device or electronicequipment that is operable to receive data. The receiver 48 may beinterchangeably referred to as a “TV converter,” “receiving device”“set-top box,” “TV receiving device,” “TV receiver,” “TV recordingdevice,” “satellite set-top box,” “satellite receiver,” “cable set-topbox,” “cable receiver,” “media player,” and/or “TV tuner.”

The headset 52 is coupled to the receiver 48. The headset 52 isconfigured to provide virtual transport of the user 22 into the virtualexperience 24 by displaying the virtual experience 24 to the user 22.Through the headset 52, the user 22 is able to participate and interactwith the virtual experience 24. By utilizing the user model data 26, itmay appear to the user 22 that they are in the event.

In some embodiments, the headset 52 may be replaced by otherpresentation devices. Examples include a TV, a personal computer (“PC”),a sound system receiver, a digital video recorder (“DVR”), a compactdisk (“CD”) device, game system, or the like. Presentation devicesemploy a display, one or more speakers, a microphone, and/or otherinput/output devices to communicate with the user 22. In manyimplementations, one or more presentation devices reside in or near theuser's 22 premises and are communicatively coupled to the receiver 48.Further, the receiver 48 and a presentation device may be integratedinto a single device. Such a single device may have the above-describedfunctionality of the receiver 48 and the headset 52, or may even haveadditional functionality.

FIG. 3 is a diagram illustrating an example of a user modeling device 54for practicing example embodiments of the virtual experience process 20according to principles disclosed herein. As shown in FIG. 3, in oneembodiment, a three dimensional virtual user model may be generated byscanning a user. The three dimensional virtual user model may begenerated by using a camera and depth sensor that use full-body 3Dmotion capture technologies to output a depth and color stream. Thedepth and color stream may then be included in the user model data 26described in reference to FIG. 1.

In some embodiments, the user modeling device 54 may also capture usermovement data 30. The user modeling device 54 may request the user 22 tomake particular motions or gestures, then the camera and depth sensorsmay record the motions and gestures to generate the user movement data30.

The user modeling device 54 may also include one or more microphones forcapturing user voice data 28. The user modeling device 54 may requestthe user 22 to speak certain words or phrases, and then record the wordsand phrases with the microphones to generate the user voice data 28.

FIG. 4 is a diagram illustrating an example of an event capture device55 for practicing example embodiments of the virtual experience process20 according to principles disclosed herein. As shown in FIG. 4, in oneembodiment, a three dimensional virtual recreation of the event may begenerated by capturing an environment with the event capture device 55.The event capture device 55 may include standard video recordingtechnology as well as the depth sensor technology to create athree-dimensional version of the event. The event capture device 55 maycapture the environment, people, objects, and sounds of an eventsimultaneously. In some embodiments, the event capture device 55 maycapture the event as multiple layers. For example, one layer may includethe background environment, and another layer may include each of thepeople or objects in the environment. The different layers may allow forreplacement of individual people and/or objects later when the virtualexperience is generated.

The data from the event capture device 55 may then be included in theenvironment data 32 and/or event data 34 described in reference toFIG. 1. In some embodiments, a user may tag the event with keywords thatdescribe the event and/or the user's experience. The keywords may thenbe used for organizing and locating specific events or experiences inthe system.

FIG. 5 is a block diagram illustrating an example of virtual merchandise56 for the virtual experience 24 according to principles disclosedherein. The virtual merchandise 56 is used to allow the user 22 to weardifferent clothes within the virtual experience 24. In some examples,the user 22 may purchase or rent the virtual merchandise 56 for use inthe virtual experience 24. In other examples, the user 22 may try on thevirtual merchandise 56 to see if the virtual merchandise fits correctlyand looks good on the three-dimensional model of the user 22 in thevirtual experience 24. In some embodiments, sampling differentmerchandise may be part of the virtual experience 24 itself. Forexample, an actor could record themselves trying on various clothes,then the user 22 could place themselves in that event and try on theexact same clothes in the same manner that the actor did. As anotherexample, the captured event may be a wedding. The user 22 may attend thewedding as a virtual experience 24 and rent a tuxedo for their virtualthree-dimensional user model to wear at the wedding.

In some examples, merchandise that was in a captured event may bereplaced with other merchandise in the virtual experience 24. Forexample, a user may capture a party event where a guest is wearing anembarrassing shirt. In the virtual experience 24 of that party, theembarrassing shirt may be replaced with another shirt. This may allowthe event to be shared as a virtual experience 24 without embarrassingthe guest wearing the embarrassing shirt.

In some examples, virtual merchandise 56 may be purchased or rented foranother user. For example, a husband may rent a pair of jeans for hiswife and see if they fit on a three-dimensional representation of her inthe virtual experience 24 before purchasing them in real-life.

FIG. 6 is a block diagram illustrating an example of virtualenvironments 58 for the virtual experience 24 according to principlesdisclosed herein. The virtual environments 58 may be examples of theenvironment data 32 described in reference to FIG. 1. As shown in FIG.5, the virtual environments 58 may include specific locations such asTimes Square in New York City, the Louvre in Paris, and the Great Wallof China, or more general locations such as a theater, a science museumor a football stadium. The user 22 may visit different virtualenvironments 58 as part of the virtual experience 24.

In some embodiments, the user 22 may mix and match different virtualenvironments 58 with different events and user models. For example, bysubstituting a captured event into another virtual environment 58, theuser 22 may create their own unique virtual experience 24. In addition,the user 22 may include different virtual merchandise 56 in the virtualexperience 24 to further customize the experience.

FIG. 7 is a schematic illustrating an example of the user 22 connectedto the virtual experience system 36 according to principles disclosedherein. In this example, the user 22 is wearing the headset 52 that isconnected to the receiver 48 through a wire connection 90. The headset52 is configured to display the virtual experience 24 to the user 22,and allow the user 22 to participate and interact within the virtualexperience 24. It should be noted that, although FIG. 6 shows that theheadset 52 is hardwired to the receiver 48, the virtual headset 52 maybe coupled to the receiver 48 wirelessly.

Although specific embodiments were described herein, the scope of theinvention is not limited to those specific embodiments. The scope of theinvention is defined by the following claims and any equivalentstherein.

What is claimed is:
 1. A virtual experience system, comprising: a serverconfigured to retrieve (a) environment data corresponding to athree-dimensional (3D) representation of a location, and (b) event datacorresponding to a captured event at the location; a receiver configuredto generate a virtual experience for a user based at least in part onthe environment data and the event data, wherein generating the virtualexperience includes: replacing a person physically present at thecaptured event with a 3D representation of the user, and replacing thelocation with a 3D representation of another location; and a displaydevice configured to display the virtual experience of the capturedevent to the user.
 2. The system of claim 1, wherein the server isfurther configured to transmit user voice data corresponding to at leastone vocal characteristic of the user to the receiver.
 3. The system ofclaim 1, wherein the server is further configured to transmit usermovement data corresponding to at least one movement characteristic ofthe user to the receiver.
 4. The system of claim 1, wherein the receiveris configured to generate the virtual experience by replacing a vocalcharacteristic of the person with a vocal characteristic of the user. 5.The system of claim 1, wherein the receiver is configured to generatethe virtual experience by replacing a movement characteristic of theperson with a movement characteristic of the user.
 6. The system ofclaim 1, wherein the virtual experience includes virtual merchandiseinserted into the virtual experience.
 7. The system of claim 6, whereinthe virtual merchandise replaces physical merchandise present at thecaptured event.
 8. The system of claim 1, wherein the captured event isassociated with a real-life event captured at the location.
 9. Thesystem of claim 1, wherein the virtual experience with the 3Drepresentation of the user inserted into the captured event is stored asa new captured event.
 10. The system of claim 1, wherein the anotherlocation is a user preferred location.
 11. The system of claim 1 furthercomprising: an event capture device that generates the event data, theevent data including multiple distinct entities, wherein a first entityof the entities is replaceable with another entity independent of theremaining entities.
 12. A method, comprising: retrieving (a) environmentdata corresponding to a three-dimensional (3D) representation of alocation, and (b) event data corresponding to a captured event at thelocation; generating a virtual experience of the captured event, for auser, based at least in part on the environment data and the event data,wherein the generating includes: replacing a person physically presentat the captured event with a 3D representation of the user, andreplacing the location with a 3D representation of another location; anddisplaying the virtual experience of the captured event to the user. 13.The method of claim 12, wherein generating the virtual experienceincludes: capturing user voice data corresponding to at least one vocalcharacteristic of the user to the receiver and user movement datacorresponding to at least one movement characteristic of the user to thereceiver.
 14. The method of claim 12, wherein generating the virtualexperience includes: replacing a vocal characteristic of the person witha vocal characteristic of the user.
 15. The method of claim 12, whereingenerating the virtual experience includes: replacing a movementcharacteristic of the person with a movement characteristic of the user.16. The method of claim 12 further comprising: inserting virtualmerchandise into the virtual experience.
 17. The method of claim 16,wherein the virtual merchandise replaces physical merchandise present atthe captured event.
 18. The method of claim 12 further comprising:storing the virtual experience with the 3D representation of the userand the location inserted into the captured event as a new event. 19.The method of claim 12, wherein the another location is a user preferredlocation.
 20. The method of claim 12, wherein retrieving the event dataincludes generating the event data as containing multiple distinctentities, wherein a first entity of the entities is replaceable withanother entity independent of the remaining entities.